The present invention is directed, in general, to video processing techniques and, more specifically, to a system and method for implementing speed-change special effects in a video stream based on the level of activity in the video content.
Speed-change special effects., such as xe2x80x9cfast forwardxe2x80x9d and xe2x80x9cslow motion,xe2x80x9d are commonly used in film and video. Both of these special effects are implemented using well-known techniques. The traditional approach for implementing a fast-forward speed change is to sub-sample the normal-speed video stream by dropping frames at a pre-determined rate, such as every other frame, every third frame, every fourth frame, every fifth frame, etc. The traditional approach for implementing a slow-motion speed change is to repeat frames in the normal-speed video stream.
Both techniques suffer significant drawbacks. In the case of a fast-forward speed change, dropping frames arbitrarily from a normal-speed stream of video frames may result in particular shots being under-represented, particularly very brief shots and shots having a high activity level. A xe2x80x9cshotxe2x80x9d is a sequence of frames that produce a single continuous scene, without a change in camera angle. xe2x80x9cActivity levelxe2x80x9d refers to the amount of change in the content of successive video frames. For example, a car chase scene or a battle scene are high-activity shots. A scene of a person sitting passively in a chair, such as a news anchorman, is an example of a low-activity shot. In the case of a slow-motion speed change, repeating frames arbitrarily in the normal-speed stream of video frames may result in a jerky motion.
There is therefore a need in the art for improved video-processing techniques for implementing speed changes in a video stream. In particular, there is a need in the art for systems and methods for performing fast-forward special effects that do not under-represent high-activity shots and shots of short duration. There is a further need for systems and methods for performing slow-motion special effects that do not result in a jerky motion in the final slow-motion shot.
To address the above-discussed deficiencies of the prior art, the present invention provides, broadly, systems and methods for altering the number of frames (i.e., adding frames or deleting frames) in a shot at rates that vary according to the activity level in the video content of the shot.
For instance, if a user selects fast forward for a particular video clip, the present invention will determine the activity level of each individual shot in the video clip and will adjust the rate at which frames are dropped in each shot according to the activity level in each shot. For example, if two persons are calmly talking in a shot, the activity level will be determined to be low and the present invention will drop frames at a comparatively high rate (e.g., drop every second frame) during fast forward. On the other hand, if two persons are engaged in a fistfight in a shot, the activity level will be determined to be high and the present invention will drop frames at a comparatively low rate (e.g., drop every fourth frame) during fast forward.
In an advantageous embodiment of the present invention, the rate at which frames are dropped may be modified xe2x80x9con-the-flyxe2x80x9d if the level of activity changes during a single shot. For example, if two person are calmly talking at the start of a shot, and then begin to fight in the middle of the shot, the activity level changes from low to high and the rate at which frames are dropped is automatically modified during fast forward to compensate for the change of activity level in the shot.
Similarly, if a user selects slow motion for a particular video clip, the present invention will determine the activity level of each individual shot in the video clip and will adjust the rate at which frames are added and xe2x80x9cmorphedxe2x80x9d in each shot according to the activity level in each shot. Morphing and other similar techniques are well known in the art and are used to reduce xe2x80x9cjerkinessxe2x80x9d in a sequence of frames. Any of these well-known techniques may be used in conjunction with the present invention to implement speed-change special effects, such as slow motion and, perhaps, fast forward.
To continue with the prior example, if two persons are calmly talking in a shot, the activity level is determined to be low and frames are added and morphed at a comparatively low rate (e.g., add one frame for each three in the original clip) during slow motion. On the other hand, if two persons are engaged in a fistfight in a shot, the activity level is determined to be high and frames are added and morphed at a comparatively high rate (e.g., add one frame for each frame in the original clip) during slow motion.
In an advantageous embodiment of the present invention, the rate at which frames are added and morphed may be modified xe2x80x9con-the-flyxe2x80x9d if the level of activity changes during a single shot. For example, if two person are calmly talking at the start of a shot, and then begin to fight in the middle of the shot, the activity level will change from low to high and the rate at which frames are added and morphed is automatically modified during slow motion to compensate for the change of activity level in the shot.
Accordingly, in one embodiment of the present invention, there is provided, for use in a video editing system, a video processing device capable of receiving a first video clip comprising at least one shot, wherein the at least one shot comprises a sequence of related frames, and modifying the video clip to perform a selected speed change special effect. The video processing device comprises an image processor capable of identifying the at least one shot and determining a first activity level within at least a first portion of the at least one shot, wherein the first activity level indicates a rate of change of video content between at least a first frame and a second frame in the at least a first portion. The image processor performs the selected speed change special effect by one of adding frames and deleting frames in the at least a first portion in response to the first activity level determination, thereby producing a modified at least one shot.
According to one embodiment of the present invention, the image processor is capable of distinguishing the at least one shot and a second shot in the first video clip and separating the at least one shot and the second shot into distinct segments prior to performing the selected speed change special effect.
According to another embodiment of the present invention, the image processor is capable of determining a second activity level within at least a first portion of the second shot and performing the selected speed change special effect by one of adding frames and deleting frames in the at least a first portion of the second shot in response to the second activity level determination, thereby producing a modified second shot.
According to still another embodiment of the present invention, the image processor is capable of combining the modified at least one shot and the modified second shot to thereby produce a modified first video clip.
According to yet another embodiment of the present invention, the selected speed change special effect is fast forward and the image processor deletes frames at a first predetermined rate if the first activity level does not exceed a first predetermined threshold.
According to a further embodiment of the present invention, the image processor deletes frames at a second predetermined rate lower than the first predetermined rate if the first activity level does exceed the first predetermined threshold.
According to a still further embodiment, of the present invention, the selected speed change special effect is slow motion and the image processor adds frames at a first predetermined rate if the first activity level does not exceed a first predetermined threshold.
According to a yet further embodiment of the present invention, the image processor adds frames at a second predetermined rate higher than the first predetermined rate if the first activity level does exceed the first predetermined threshold.
The foregoing has outlined rather broadly the features and technical advantages of the present invention so that those skilled in the art may better understand the detailed description of the invention that follows. Additional features and advantages of the invention will be described hereinafter that form the subject of the claims of the invention. Those skilled in the art should appreciate that they may readily use the conception and the specific embodiment disclosed as a basis for modifying or designing other structures for carrying out the same purposes of the present invention. Those skilled in the art should also realize that such equivalent constructions do not depart from the spirit and scope of the invention in its broadest form.
Before undertaking the DETAILED DESCRIPTION, it may be advantageous to set forth definitions of certain words and phrases used throughout this patent document: the terms xe2x80x9cincludexe2x80x9d and xe2x80x9ccomprise,xe2x80x9d as well as derivatives thereof, mean inclusion without limitation; the term xe2x80x9cor,xe2x80x9d is inclusive, meaning and/or; the phrases xe2x80x9cassociated withxe2x80x9d and xe2x80x9cassociated therewith,xe2x80x9d as well as derivatives thereof, may mean to include, be included within, interconnect with, contain, be contained within, connect to or with, couple to or with, be communicable with, cooperate with, interleave, juxtapose, be proximate to, be bound to or with, have, have a property of, or the like; and the term xe2x80x9ccontrollerxe2x80x9d or xe2x80x9cprocessorxe2x80x9d means any device, system or part thereof that controls at least one operation, such a device may be implemented in hardware, firmware or software, or some combination of at least two of the same. It should be noted that the functionality associated with any particular controller or processor may be centralized or distributed, whether locally or remotely. Definitions for certain words and phrases are provided throughout this patent document, those of ordinary skill in the art should understand that in many, if not most instances, such definitions apply to prior, as well as future uses of such defined words and phrases.